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ABSTRACT 
This report presents a method for analyzing 

curriculum using the examinations of the National Board of Medical 
Examiners (NBME). The method provides a system by which the NBME 
examinations can be used each year to compare the content of an 
external licensing test with the content of the curriculum at the 
Wright State University School of Medicine (WSUSOM). The NBME test 
items were classified by faculty content experts on four dimensions: 
school-wide topics, discipline topics, cognitive levels, and item 
types. Then, the percent of WSUSOM students and percent of national 
sample answering the item correctly were recorded. The analysis 
allows comparison of the performance of WSUSOM student and.the 
national sample on each school-wide or discipline topic. The Faculty 
Curriculum Committee is the most interested user of data related to 
school-wide topics, while the individual department is the most 
appropriate audience for the discipline topic information. Curriculum 
decision-makers should take special caution in examining the results 
for a single year's administration, since test content varies from 
year to year. (Author/BW) 
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Purpose 
& 


The examinations of the National Board of Medical Examinations (NBME) 
é 
are used extensively by U.S. medical schools to evaluate institutional 
effectiveness. The 1980 Annual Report of the National Board reported that 


, about 75 percent of U.S. medical schools use the NBME exams for institu- 


tional assessment. Despite their program evaluation importance to medical 


achools, very little has been reported in the literature to guide local 
efforts in making the best use of the NBME examinations. 

This report presents a method for analyzing curriculum using the 
examinations of the NBME. The method provides a system by which the NBME 


‘y examinations can be used each year to compare the content of an external 
s 
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licensing test with the content of the WSUSOM curriculum. The analyses 
provide data to medical school curriculum decision-makers which can be 
used to examine topics which cut acrosa individualé disciplines. Multi- 
disciplinary information of this kind permit a cooperative effort within 
the medical school i evaluating teaching effectiveness. In addition, 
departments are provided with data on the content areas identified as 
specific to their disciplines. 

Review of the Literature 

Curriculum evaluation using the National Boards has most frequently 
used the NBME Patt I (Sanazaro, 1967; Kennedy et al., 1970; Wile, 1978; 
Garrard et al., 1978; McGaghie et al., 1980; Mcbaghie et al., 1981; Baum 
et al., 1981). Studies using Part I generally have focused on (1) the 
match between NBME content and local content (Kennedy et al., 1970; Wile, 
1978; Garrard et al., 1978; McGaghie et al., 1980; McGaghie et al., 1981) 
and (2) the relationship between local curriculum emphasis allotted to 

\ 
NBME content and student achievement (Sanazaro, 1967; Wile, 1970; Garrard 
et al., 1978; McGaghie et al., 1980; McGaghie et es: 1981). 

While evaluation studies using Part I have examined all or nearly all 
of the component disciplines, Part II investigations have focused on one 
individual discipline: surgery (Linn et al., 1979; Benenson et al., 1981), 
obstetrics and gynecology (Spellacy and Dockery, 1980), and medicine 
(Calhoun et al., 1980). The validity of local Svatistdon measures was 
the focus of three of these studies (Linn et al., 1979; Spellacy and 
Dockery, 1980; Benenson et al., 1981). The Calhoun et al. study (1980 ) 


examined the relationship between stude achievement and type of 


clerkship activity (patient care, joint pattent care and 


education). Pe ne 
awe 


Oe ce 


Method 

As a response to a mandate from the Wright State Uiivecnley School of 
Medicine Faculty Curriculum Committee, the authors formlated a plan to ~ 
analyze the test items on the Part I and II examinations of the NBME. 
Part I was analyzed following its administration in June 1981 and Part II 


following ite administration in September 1981. This first round of 


analysis in 1981 will be followed each succeeding year by similar 
investigations. 

For Part I one of the authors (SNK) classified the items for the 
seven examinations (anatomy, behaviorial science, biochemistry, micro- 
biology, pathology, pharmacology and physiology). For Part II a.WSUSOM 
faculty content expert from each of the six disciplines (medicine, 


obstetrics/gynecology, pediatrics, preventive medicine/public health, 


al 


ails and surgery) classified the items. WSUSOM faculty content 
experts were not used for Part I because this first investigation was 
“~ “éonsidered a pilot study of a new methodology. However, all succeeding 
analyses will employ WSUSOM faculty content experts for both Part I and 
Part II. 
- 


For the 1981 round of analysis Table 1 shows the four dimensions on 


which each item was classified and the variation in classification systems 
oer in Part I and Part II. A more standardized classification systen, 
where appropriate, will be developed for the 1982 investigations. For 
Part II in 1981 (and for all succeeding analyses of Part I and Part Il), 
eonkent experts were permitted.to use more than one category in classi- 


fying items by school-wide topic and discipline topic. Multiple 


classifications permit a better understanding of an exam's content. 


Tabli: 1: Classification of Items on NBME Part I and Part II 


Part II 


School-wide topics 23 33 


Discipline topics varies BY varies 


Cognitive levels 2a 3b 


Icem types 5¢ 4d 


8factual recognition/knowledge and application of knowledge/ 
problem-solving - 

bfactual recognition/knowledge, diagnosis, treatment /management 
Cone best response, one best response (except), matching list, 
matching compare/contrast, mltiple true/false 

done test response, matching list, matching compare/contrast , 

miltiple true/false 


Finally, the perceftt: of WSUSOM students and percent of the national 
sample answering the item correctly were recorded. This permitted each 
item to be categorized on the haute of WSUSOM versus national performance 
with intervals of 5 percentage points used to form cells. Thus, when 
-WSUSOM and the national sample differed by 5 percentage points or less on 
an item, the item was classified as same (i.e., no difference between WSUSOM 
and the national sample). A difference of greater than 5 percentage points 
resulted in an item being placed either in the WSUSOM or nation column. 

A few examples from the 1981 Part I and Part II analysis should 
suffice to demonstrate how curriculum decision-makers can examine content 
which cuts across divctp lines, School-wide Topic A had 119 items on Part I 


and 27 items on Part II. On about one-half (55) of the part I items, 


WSUSOM students scored the same as the national sample. However, on the 
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remaining 64 Part I items, WSUSOM students were lower on 47 (73%) of the 
items. Performance on School-wide Topic A did not improve on Part IIL. 
While two-thirds of the 27 items (18) were in the same column, WSUSOM was 
outscored on 8 of the 9 remaining items. School-wide Topic A is an 
example of low WSUSOM performance on both National Boards Part I and Part 
II. There were 35 School-wide Topic B items on Part I and 19 School-wide 
Topic B items on Part II. WSUSOM scored the same as the national sample 
on 16 Part I items, better on 11 items, and worse on 8 items. Serre 
WSUSOM scored the same on 10 Part II items, better on 7 items and worse 
on 2 items. School-wide Topic B illustrates content on which WSUSOM 
scored better than the national sample on both Part I and Part II. ? 

Other school-wide topics can be found on which WSUSOM performance 
was better (or worse) than the national sample on both Part I and Part II. 
In addition, there are school-wide topics on which WSUSOM Part I perfor- 
mance was high while Part II performance was low, and vice versa. School- 
wide Topic C had WSUSOM outperforming the national sample on a 2 to l 
basis on those items where there was a difference on Part I. ere on 
Part II the national sample outperformed WSUSOM by a small margin. In 
contrast, the national sample did better by more than a 3 to 1 ratio on 
School-wide Topic D for Part I, but WSUSOM reversed the results on Part 
II with a 2 to 1 ratio of better performance on items where there was a 
difference. 

In addition to comparison from Part I to Part II, faculty ean examine 
the disciplines which a most influential in producing a positive (or 
negative) result. For example, was the poor performance on School-wide 


Topic A due to a few of the eleven disciplines included or were the low 


results umiform across disciplines. Similarly, was high achievement on 
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School-wide Topic B due to a few or most disciplines. Finally, the second 
type of content classification-within departments-can be beneficial to 
faculty. For the 1981 analysis each department classified items according 
to its view of how its discipline is organized. Examination of the 
results from this analysis can be helpful to departmental faculty in 
evaluating discipline topics as they are taught in the curriculum. 
Discussion 

The method presented in this report provides information to 
curriculum decision-makers such as the Faculty Curriculum Committee, 
department chairmen, and course directors through evaluation of NBME 
examinations. The Faculty Curriculum Committee is the most interested user 
of data related to school-wide topics while the individual department is 
the most appropriate audience for the discipline topic information. 
Curriculum decision-makers: are advised to take special caution in examining 
the results for a single year's administration. The content of NBME exams 
varies from administration to administration, and only through the 
compilation of results over a number of years can validity be attached 
to the findings. The system presented in this report will be employed 
each year to analyze NBME Part I and Part II examinations with the intent 
of providing curriculum decision-makers with valuable data on which to 


make judgments. 
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